Defect image generation method for deep learning and system therefor

ABSTRACT

A method of generating a defect image for deep learning and a system therefor are provided. The method and the system are intended to be used in generating training data for an artificial intelligence algorithm. More specifically, the training data are defect images required to train an algorithm that identifies a defect from a product.

CROSS REFERENCE TO RELATED APPLICATION

This application claims priority to Korean Patent Application No.10-2020-0135737, filed on Oct. 20, 2020, the disclosure of which isincorporated herein by this reference in its entirety.

FIELD

Methods and apparatuses consistent with exemplary embodiments relate togenerating a defect image for deep learning and a system therefor.Particularly, the invention relates to generating a defect image fortraining an artificial intelligence algorithm to identify a defect in aninspection image.

BACKGROUND

There are several methodologies for identifying whether a defect existsin a product. A representative method for identifying the existence of adefect is to acquire an image of a product and analyze the acquiredimage.

Recently, artificial intelligence algorithms have been widely used fordata analysis tasks, particularly, for an analysis requiring a lot ofcomputation with big amounts of data. An artificial intelligencealgorithm may be used to detect defections. It is expected that if theartificial intelligence algorithm was appropriately trained, theaccuracy of defect detection of the artificial intelligence algorithmwould be as effective as human intellectual judgement.

In order to enhance the accuracy of the artificial intelligencealgorithm, an appropriate training needs to be preceded. In general,various types and huge volume of training data are required for anappropriate training. However, it is not easy to generate or collectsuch training data in practice.

In related art, common image augmentation methods were used forproducing training data, such as, rotating, flipping, rescaling,resizing, shearing, zooming, and adding a noise. However, such methodscreate only a simple transformation of a defect (fault) image. They arelimited in creating a new form of defect (fault) images that can besupplied to the artificial intelligence algorithm.

Accordingly, the demand for producing training data enabling theartificial intelligence algorithm to be sufficiently trained has beengradually increased. In particular, it is required to produce trainingimage data that allow to detect various types and shapes of defects in aproduct. The present disclosure is proposed in view of thecircumstances.

The description presented herein is to solve the technical problemsmentioned above as well as to provide additional technical elements thatare not easily invented by those skilled in the art.

BRIEF DESCRIPTION

The present disclosure is directed to generating various types of defectimages for training and, furthermore, to quickly generating many defectimages.

In addition, this invention is directed to enhancing the accuracy ofdetecting a defect in an inspection image by training an artificialintelligence algorithm with the generated defect images.

It is to be understood that technical problems to be solved by thepresent disclosure are not limited to the aforementioned technicalproblems, and additional aspects will be set forth in part in thedescription which follows and, in part, will become apparent from thedescription.

According to an embodiment of the present disclosure, there is provideda method of generating a defect image for training, the methodincluding: extracting a defect area from a sample image; determining atarget area in a base image, the target area being an area with whichthe defect area is to be synthesized; correcting the defect area byreferring to image data of the target area; and generating the defectimage for training by synthesizing the corrected defect area with thetarget area in the base image.

In addition, the method of generating the defect image for learning mayfurther include, after the extracting of the defect area, transforming ashape of a defect in the defect area.

In addition, in the method of generating the defect image for learning,the determining the target area in the base image comprises using theshape of the transformed defect and a surrounding area around thetransformed defect. Herein, either or both of a position and a size ofthe target area in the base image is randomly determined.

In addition, in the method of generating the defect image for learning,a correction for reducing a difference in image data between asurrounding area in the defect area and a surrounding area in the targetarea is made.

In addition, in the method of generating the defect image for learning,a histogram correction of either or both of the surrounding area in thedefect area and the surrounding area in the target area is made.

In addition, in the method of generating the defect image for learning,the generating of the defect image for training by synthesizing thecorrected defect area with the target area in the target image comprisesperforming an image adjustment on the defect area.

In addition, in the method of generating the defect image for learning,the performing of the image adjustment comprises performing a bluroperation on an edge of either or both of a defect and a surroundingarea of the defect.

According to another embodiment of the present disclosure, there isprovided a method of generating a defect image for training, the methodincluding: extracting a defect area from a sample image; transforming ashape of the defect area; and generating the defect image for trainingby synthesizing the defect area with a base image.

In addition, in the method of generating the defect image for learning,the generating of the defect image for training comprises inserting thetransformed defect area to a target area where the position of thetarget area is determined at random.

In addition, in the method of generating the defect image for learning,the generating of the defect image for training comprises performing animage adjustment on the defect area after the defect area is inserted tothe target area.

According to still another embodiment of the present disclosure, thereis provided a system for generating a defect image for learning, thesystem including: an extraction component extracting a defect area froma sample image; a target area determination component determining atarget area in a base image, the target area being an area with whichthe defect area is to be synthesized; a correction component correctingthe defect area by referring to image data of the target area; and asynthesis component generating the defect image for training bysynthesizing the corrected defect area with the target area in the baseimage.

In addition, the system for generating the defect image for learning mayfurther include a transformation component transforming a shape of adefect in the defect area.

According to the present disclosure, various types of defect images fortraining can be quickly and easily generated to provide a sufficientamount of training data.

In addition, according to the present disclosure, the various types ofdefect images enable a sufficient training of the artificialintelligence algorithm which results in an enhanced accuracy ofdetecting a defect.

Effects that may be obtained from the present disclosure are not limitedto the aforementioned effects. Other effects which are not describedherein will be clearly understood by those skilled in the art from thefollowing description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1 and 2 are diagrams illustrating a function and the reason fortraining an artificial intelligence algorithm described in the presentdisclosure;

FIG. 3 is a flowchart for generating a defect image for trainingaccording to a first embodiment;

FIG. 4 is a diagram for better understanding of the method of generatingthe defect image for training according to the first embodiment;

FIG. 5 is a flowchart for generating a defect image for trainingaccording to a second embodiment; and

FIG. 6 is a diagram for better understanding of the method of generatingthe defect image for training according to the second embodiment.

DETAILED DESCRIPTION

Various modifications and various embodiments will be described indetail with reference to the accompanying drawings so that those skilledin the art can easily carry out the disclosure.

It should be understood, however, that the various embodiments are notfor limiting the scope of the disclosure to the specific embodiment, butthey should be interpreted to include all modifications, equivalents,and alternatives of the embodiments included within the spirit and scopedisclosed herein.

In addition, although one or more functional blocks of the presentdisclosure are expressed as individual blocks, one or more of thefunctional blocks of the present disclosure may be a combination ofvarious hardware and software components executing the same function.

The expression “including elements” is an open-ended expression whichmerely refers to existence of the elements, and it should not beconstrued as excluding additional elements.

It should be understood that when an element is referred to as being“coupled to” or “connected” to another element, it can be directlycoupled or connected to the other element or intervening elements may bepresent therebetween.

Hereinafter, a method of generating a defect image for training and asystem therefor will be described.

FIG. 1 briefly describes detecting a defect using an artificialintelligence algorithm according to an exemplary embodiment. Theartificial intelligence algorithm is supplied with an image forinspection, determines whether a defect exists in the image, and detectsa defect area.

The image for inspection means image data generated by photographing aproduct. Herein, there is no particular limit on the product to beobserved. The artificial intelligence algorithm in the presentdisclosure may be understood as a kind of non-destructive inspection.For example, a defect may be detected by performing an image analysis ona transmission image, such as an X-ray image. As long as the input tothe artificial intelligence algorithm has the form of any image data,the artificial intelligence algorithm is able to identify and detect adefect.

The aforementioned artificial intelligence algorithm may be executed bya device having a central processing unit (CPU) and a memory. Herein,there is a premise that the artificial intelligence algorithm isimplemented to enable machine learning, more precisely, deep learning.

FIG. 2 is a diagram conceptually illustrating a process of training anartificial intelligence algorithm with multiple training images toenhance the performance of a model. Deep learning can be defined as amachine learning technique that teaches computers to learn by example,and it can be used predict y from x when there is no discerniblerelationship between x and y. There are various types of deep learningmethods. A method of building a model by stacking neural networks inmultiple layers is understood as deep learning.

Representative deep learning models include a deep neural network, aconvolutional neural network, a recurrent neural network, and arestricted Boltzmann machine. In the deep neural network, multiplehidden layers exist between an input layer and an output layer. Theconvolutional neural network forms the connectivity pattern betweenneurons which resembles the organization of animal visual cortex. Therecurrent neural network builds up a neural network every moment overtime. The restricted Boltzmann machine may learn a probabilitydistribution over sets of inputs.

The artificial intelligence algorithms capable of such deep learningbasically go through a process of training because it is difficult forthe artificial intelligence algorithms to learn causality on their ownwithout any information given at first. The learning process requires alot of computation with big amounts of data. In general, computing powercan be scaled up as needed, but there is a limit on the amount oftraining data that humans can provide. Insufficient number of trainingdata greatly reduces the performance of the artificial intelligencealgorithms. This invention is intended to be used to produce varioustypes and an enormous amount of training data for an artificialintelligence algorithm. To put it simply, the present disclosure relatesto a method and a system for generating training data shown on the leftin FIG. 2.

FIG. 3 is the flowchart for generating a defect image for trainingaccording to a first embodiment of the present disclosure. The methodmay be performed by a system having a central processing unit and amemory. Herein, the central processing unit may also be called acontroller, a microcontroller, a microprocessor, or a microcomputer. Inaddition, the central processing unit may be implemented as hardware,firmware, software, or a combination thereof. When the centralprocessing unit is implemented using hardware, the hardware may beconfigured to include an application specific integrated circuit (ASIC),a digital signal processor (DSP), a digital signal processing device(DSPD), a programmable logic device (PLD), or a field programmable gatearray (FPGA). When the central processing unit is implemented usingfirmware or software, the firmware or software may be configured toinclude a module, procedure, or function for performing theabove-described functions or operations. In addition, the memory may berealized as a read-only memory (ROM), a random-access memory (RAM), anerasable programmable read-only memory (EPROM), an electrically erasableprogrammable read-only memory (EEPROM), a flash memory, a static RAM(SRAM), a hard disk drive (HDD), or a solid state drive (SSD).

According to the first embodiment, the method of generating the defectimage for training may include a process of extracting a defect, aprocess of transform the defect, a process of correcting a defect areaincluding the defect, and a process of synthesizing the defect area witha base image. That is, the method may include a process of extractingany defect from a sample image, performing a transformation of thedefect in various ways, and performing a synthesis of the transformeddefect and the base image. Hereinafter, the method of generating thedefect image for training according to the first embodiment will bedescribed in more detail with reference to FIGS. 3 and 4.

The method of generating the defect image for training may begin withstep S101 of extracting a defect area from a sample image. The sampleimage means an image acquired by photographing a product having adefect. The sample image may be acquired by photographing a producthaving a defect in a manufacturing process, an image acquired during amaintenance of a product, or an image acquired after intentionallymaking a defect on a product. In addition, according to the presentdisclosure, a large number of defect images with various shapes may begenerated only from a sample image by generating multiple defect images.

The defect area may be categorized into two types. First, the defectarea may be defined as an area including only a defect itself. That is,the inside of the closed curve formed by the outline of the defect is adefect area if a defect existed on a product. Herein, various analysismethods utilizing image data may be used to extract the defect area,such as boundary line extraction using differences in RGB (Red, Green,Blue) values or HSV (Hue, Saturation, Lightness) values between pixelsconstituting the image.

Second, the defect area may include an actual defect as well as apredetermined area surrounding the defect. That is, the defect and thesurrounding area around the defect constitute a defect area. Herein, thedefect area may be determined by a preset condition. For example, thecondition may be set such that when the actual defect is identified, thedefect area is the inside of the quadrangle defined by drawinghorizontal and vertical lines on the outermost points of the top,bottom, left, and right of the defect.

Additionally, the defect area may be determined by a user input on thesample image. For example, on the system in which a sample image isloaded, a user designates an area surrounding any defect through a mouseinput, a touch pen input, or a touch input to determine a defect area.Herein, the user defined defect area may include only the actual defect,or the surrounding area around the defect. As will be described later,the actual defect of the defect area is for generating a defect of a newshape, and the surrounding area is for generating a synthetic image freefrom unnatural feeling.

Referring to FIG. 4, the upper left of the figure shows a process ofextracting a defect area 20 from a sample image 10. A defect may existat any position on the sample image 10. Any part of the sample image, aslong as a defect is included, can be identified and extracted as adefect area 20. Herein, the defect area may include the inside of theclosed curve of the defect, or may include both the defect itself and apredetermined surrounding area.

Referring to FIG. 3, after step S101, step S102 of transforming thedefect may be executed. Step S102 is for generating defects in variousshapes. Shape transformations, such as changing a size of a defect,transforming an outline shape of a defect, rotating a defect, orflipping a defect, may be performed to transform the defect shape. Ineach of the transformations, the degree of transformation is randomlydetermined to generate a large number of various types of defect shapes.

Step S102 may be an optional step in the method of generating the defectimage for training, and performed only when necessary. If step S102 isomitted, a final defect image generated for training may be the sameshape as the defect area 20 or the defect extracted from the sampleimage 10.

After step S102, step S103 of determining a target area in the baseimage may be executed. The base image may be understood as thebackground image of a final defect image, or an image that issynthesized with the defect extracted from the sample image. The targetarea may be understood as an area with which the defect area or defectextracted from the sample image is to be synthesized. Therefore, stepS103 may be understood as a step of determining which position on thebase image the defect area is to be synthesized with.

At step S103, a previously extracted defect area or defect may bereferenced in the process of determining the target area. For example,the size, length, and width of the defect area, the shape, length, andwidth of the defect, or the shape, length, and width of the defect afterperforming transformation may be referenced in determining the targetarea. The target area may be determined to have the same size or shapeas that of the defect area (or actual defect) to avoid an error in imagesynthesis due to the difference between the defect area (or actualdefect) and the target area. For reference, at least one attributebetween the position or the size of the target area in the base imagemay be determined randomly. That is, information acquired from thepreviously extracted defect area may be referenced in determining thetarget area, but either or both of the position and the size of thetarget area may be determined randomly.

Referring to FIG. 4, the upper right shows the process of determiningthe target area 40 in the base image 30. Herein, it is found that thedefect area 20 or defect extracted from the sample image 10 isreferenced for the target area 40.

After step S103, step S104 of correcting the defect area may beexecuted. Step S104 may be understood as a process of minimizingunnatural feeling from a synthetic image. If the defect area or defectextracted from the sample image was inserted to the base image as it is,the final output image might not look like a normal defect image, whichresults in loss of quality of the training data. The step ofpre-correcting the defect area is provided to make a defect image asseamless as possible.

The two images on the center of FIG. 4 labeled as Correct show a processof correcting the defect area 20 by contrasting the defect area 20 withthe target area 40. Among various correction methods, histogramcorrection may be an optimal choice. A histogram means a graph generatedwith variables, wherein an intensity value of an image is plotted on thehorizontal axis and the number of pixels corresponding to the intensitylevel is plotted on the vertical axis. The graph allows to determine howbright or dark an image is. At step S104, the light and darkness of thedefect area 20 may be corrected by measuring the number of pixels ofeach intensity values of the target area 40 in the base image 30. Thatis, a correction may be made to move the graph corresponding to thedefect area 20 on the left in the histogram toward the graphcorresponding to the target area 40.

It has been described above that a defect area may be defined in twotypes. The first is to include only a defect itself. The second is toinclude a defect and a surrounding area. The correction process at stepS104 may be performed in both cases. In case of the first type, thecolor of the defect area is corrected to set histogram values similar tothose of the target area. Herein, the degree of correction may berestricted to only allow the histogram of the defect close to thehistogram of the target area 40, but not equal. It may be difficult todistinguish the defect area 20 if the histogram of the corrected imageis same as the histogram of the target area 40. Therefore, in case ofthe first type, correction may be limited to a certain degree that wouldnot make the histogram of the defect area equal to the histogram of thetarget area 40. In case of the second type, an additional correction ofthe surrounding area may be made. In some cases, a correction of theactual defect may be omitted and only a correction of the surroundingarea may be made. Regarding the correction of the surrounding area ofthe defect, a histogram correction may be made similar to that of thetarget area. One way to minimize unnatural feeling from a syntheticimage is to make the surroundings look seamless by correcting thesurroundings of the area that the defect is being positioned. To thisend, a histogram correction for reducing the difference between thesurrounding area of the defect and the target area (or the surroundingarea of the target area) may be made.

After step S104, step S105 of generating a defect image for training bysynthesizing the corrected defect area with the target area in the baseimage may be executed. Step S105 may be understood as a step of addingthe defect area created at the previous step to the base image. The stepof generating the defect image for training may include an adjustment onthe defect area to make the defect image more seamless. Various methodsmay be applied for the adjustment, including blur operation,interpolation operation, or harmonization operation on an edge of eitheror both of the defect and surrounding area of the defect.

The bottom of FIG. 4 shows the defect area that the transformed shape issynthesized with the target image which is a generated defect image 50.

The method of generating the defect image for training according to thefirst embodiment of the present disclosure has been described withreference to FIGS. 3 and 4.

FIGS. 5 and 6 are diagrams illustrating a method of generating a defectimage for training according to a second embodiment of the presentdisclosure. Unlike the first embodiment, according to the secondembodiment, a defect area is extracted from a sample image, the shape ofthe defect area is transformed, and the defect area is directlysynthesized with a base image. That is, the process of correcting thedefect area is omitted. Instead, the defect area is synthesized with thebase image, and a correction follows to generate a defect image fortraining.

Referring to FIG. 5, the method of generating a defect image fortraining according to the second embodiment may begin with step S201 ofextracting a defect area from a sample image. The notion of a sampleimage and defect area are same as those in the first embodiment exceptthe defect area in the second embodiment only refers to a defect itself.That is, the defect area may be the inside of the closed curve formed bythe outline of the defect.

After step S201, step S202 of transforming the shape of the defect maybe executed. Step S202 is for generating defect images in various shapesby a shape transformation techniques including changing a size of adefect, transforming an outline shape of a defect, rotating a defect, orflipping a defect.

Immediately after step S202, step S203 of generating a defect image fortraining by synthesizing the transformed defect with the target imagemay be executed. At step S203, an image adjustment on the defect areamay be further included after the transformed defect is inserted to thetarget area.

The image adjustment is a process of making the training image data asseamless as possible. When performing the image adjustment, as describedin the first embodiment, the following methods may be applied: bluroperation, interpolation operation, or harmonization operation on anedge of the transformed defect area. The adjustment of an image mayfurther include the surrounding area of the transformed defect area toreduce unnatural feeling caused by the synthesis. This process may beseparately performed.

FIG. 6 shows the method of generating a defect image for trainingaccording to the second embodiment. The method generates an image in thefollowing order: extracting a defect from a sample image 10 andtransforming the defect 21; synthesizing the transformed defect 41 withthe base image 30; and performing image adjustment on the target area 40or transformed defect 41 in the base image.

The method of generating the defect image for training according to thesecond embodiment has been described with reference to FIGS. 5 and 6.

The first and the second embodiment described above are methodsperformed by a computing device having a central processing unit and amemory. A system in which each embodiment may be implemented will bedescribed as follows.

First, a system 100 corresponding to the first embodiment may include:an extraction component 110 extracting a defect area from a sampleimage; a target area determination component 130 determining a targetarea in a base image where the target area is an area with which thedefect area is to be synthesized; a correction component 140 correctingthe defect area by referring to image data of the target area; and asynthesis component 150 generating a defect image for training bysynthesizing the defect area corrected by the correction component withthe target area. In addition, the system 100 may further include atransformation component 120 for transforming a shape of a defect in thedefect area.

Second, a system 200 corresponding to the second embodiment may include:an extraction component 210 extracting a defect area from a sampleimage; a transformation component 220 transforming a shape of a defectin the defect area; and a synthesis component 230 generating a defectimage for training by synthesizing the transformed defect with the baseimage.

A method of generating a defect image for training and a system thereforhave been described. While exemplary embodiments have been describedwith reference to the accompanying drawings, it will be apparent tothose skilled in the art that various modifications in form and detailsmay be made therein without departing from the spirit and scope asdefined in the appended claims. Therefore, the description of theexemplary embodiments should be construed in a descriptive sense and notto limit the scope of the claims, and many alternatives, modifications,and variations will be apparent to those skilled in the art.

What is claimed is:
 1. A method of generating a defect image fortraining, by a system having a central processing unit and a memory, themethod comprising: extracting a defect area from a sample image;determining a target area in a base image, the target area being an areawith which the defect area is to be synthesized; correcting the defectarea by referring to image data of the target area; and generating thedefect image for training by synthesizing the corrected defect area withthe target area in the base image.
 2. The method of claim 1, furthercomprising: after the extracting of the defect area, transforming ashape of a defect in the defect area.
 3. The method of claim 2, whereinthe determining the target area in the base image comprises using theshape of the transformed defect and a surrounding area around thetransformed defect.
 4. The method of claim 3, wherein either or both ofa position and a size of the target area in the base image is randomlydetermined.
 5. The method of claim 1, wherein the correcting of thedefect area comprises a correction for reducing a difference in imagedata between a surrounding area in the defect area and a surroundingarea in the target area.
 6. The method of claim 5, wherein thecorrection for reducing a difference comprises a histogram correction ofeither or both of the surrounding area in the defect area and thesurrounding area in the target area.
 7. The method of claim 6, whereinthe histogram correction does not make a histogram of the defect areaequal to a histogram of the target area.
 8. The method of claim 1,wherein the generating of the defect image for training by synthesizingthe corrected defect area with the target area in the target imagecomprises performing an image adjustment on the defect area.
 9. Themethod of claim 8, wherein the performing of the image adjustmentcomprises performing a blur operation on an edge of either or both of adefect and a surrounding area of the defect.
 10. A method of generatinga defect image for training, by a system having a central processingunit and a memory, the method comprising: extracting a defect area froma sample image; transforming a shape of the defect area; and generatingthe defect image for training by synthesizing the defect area with abase image.
 11. The method of claim 10, wherein the generating of thedefect image for training comprises inserting the transformed defectarea to a target area where the position of the target area isdetermined at random.
 12. The method of claim 11, wherein the generatingof the defect image for training comprises performing an imageadjustment on the defect area after the defect area is inserted to thetarget area.
 13. A system for generating a defect image for training,wherein the system has a central processing unit and a memory, thesystem comprising: an extraction component extracting a defect area froma sample image; a target area determination component determining atarget area in a base image, the target area being an area with whichthe defect area is to be synthesized; a correction component correctingthe defect area by referring to image data of the target area; and asynthesis component generating the defect image for training bysynthesizing the corrected defect area with the target area in the baseimage.
 14. The system of claim 13, further comprising: a transformationcomponent transforming a shape of a defect in the defect area.
 15. Thesystem of claim 13, wherein the target area determination componentdetermines the target area using the shape of the transformed defect anda surrounding area around the transformed defect.
 16. The system ofclaim 15, wherein either or both of a position and a size of the targetarea in the base image is randomly determined.
 17. The system of claim13, wherein the correction component performs a correction for reducinga difference in image data between a surrounding area in the defect areaand a surrounding area in the target area.
 18. The system of claim 17,wherein the correction component performs a histogram correction ofeither or both of the surrounding area in the defect area and thesurrounding area in the target area.
 19. The system of claim 18, whereinthe correction component does not make a histogram of the defect areaequal to a histogram of the target area.
 20. The system of claim 13,wherein the synthesis component performs an image adjustment on thedefect area by performing a blur operation on an edge of either or botha defect or a surrounding area of the defect.